Search Engine / / 2023. 4. 7. 14:09

Elasticsearch & Kibana

1. 개요

* 기존 ELK( Elasticsearch, Logstash, Kibana )

* NiFi 를 이용하여 Data Lake 를 구축 - DFM( Data Flow Management )

* nar 파일을 이용하여 컴포넌트 생성

* 설치가 반이다.

- Elastic Search 는 러닝커버가 제로!

- Elastic Search * 10 > Sola // -> 구글기반에서 많이 사용됨

- Apache Lucene 기반의 JAVA 오픈소스 분산 검색엔진

- 역인덱스, 데이터를 컬럼을 모두 쪼개어서 단어에 인덱스를 부여하고 문서를 매핑시킴

- scoreing, bool, should (포함되었으면 좋겠는), must(반드시 포함되어야하는), ..._not

- 쿼리에서 우선순위를 결정할 수 있음

- Java REST Client 인터페이스 생성시 JAVA는 업그레이드가 빈번히 발생하고 하위호환이 되지 않아 어려움

- Python이나 Node를 주로 사용하여 인터페이스 생성

- 검색앤진에서는 INSERT가 아닌 INDEXING으로 표현한다.

- nori analizer (ex : 은전한닢... 등), 파이썬의 Mecab 라이브러리 사용

- TFIDF : 단어의 빈도수

2. 수행 목차

- 개발환경 : EC2 Linux Ubuntu 20.04.6 LTS
- Elastic Search 8.7.0 설치
- Elastic Search 서비스 등록 (재부팅시 자동실행)
- Kibana 8.7.0 설치
- Kibana 서비스 등록 (재부팅시 자동실행)
- 접속 테스트

3. 수행

Elastic Search 8.7.0 설치

1. JDK 17 이상으로 설치

2. Elastic Search 8.7.0 설치 (https://www.elastic.co/kr/downloads/past-releases/elasticsearch-8-7-0)
3. 설정파일 수정 후 실행

$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.7.0-linux-x86_64.tar.gz
$ tar -zxvf elasticsearch-8.7.0-linux-x86_64.tar.gz
$ cd elasticsearch-8.7.0

$ vi /config/jvm.options

-Djava.net.preferIPv4Stack=true  # JVM 옵션란에 추가

$ ./bin/elasticsearch

---> 설정파일 자동구성됨
# 로그에서
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Elasticsearch security features have been automatically configured!
✅ Authentication is enabled and cluster connections are encrypted.

ℹ️  Password for the elastic user (reset with `bin/elasticsearch-reset-password -u elastic`):
  ***** #elastic password

ℹ️  HTTP CA certificate SHA-256 fingerprint:
  *****

ℹ️  Configure Kibana to use this cluster:
• Run Kibana and click the configuration link in the terminal when Kibana starts.
• Copy the following enrollment token and paste it into Kibana in your browser (valid for the next 30 minutes):
  ***** # enrollment token

ℹ️ Configure other nodes to join this cluster:
• Copy the following enrollment token and start new Elasticsearch nodes with `bin/elasticsearch --enrollment-token <token>` (valid for the next 30 minutes):
  ***** # enrollment token

  If you're running in Docker, copy the enrollment token and run:
  `docker run -e "ENROLLMENT_TOKEN=<token>" docker.elastic.co/elasticsearch/elasticsearch:8.7.0`
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

# 비밀번호 재설정 (필요시)
$ ./bin/elasticsearch-reset-password elastic -i

Elastic Search 서비스 등록

$ adduser elastic
$ chown -R elastic:elastic /home/ubuntu/elasticsearch-8.7.0
$ sudo vi /etc/systemd/system/elasticsearch.service
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[Unit]
Description=elasticsearch server
Wants=syslog.target network.target network-online.target
After=syslog.target network.target network-online.target

[Service]
RuntimeDirectory=elasticsearch-8.7.0
WorkingDirectory=/home/ubuntu/elasticsearch-8.7.0

LimitMEMLOCK=infinity
LimitNOFILE=65535
LimitNPROC=4096

ExecStart=/home/ubuntu/elasticsearch-8.7.0/bin/elasticsearch
ExecReload=/home/ubuntu/elasticsearch-8.7.0/bin/elasticsearch
RestartSec=3s

User=elastic
Group=elastic

[Install]
WantedBy=multi-user.target
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
$ sudo systemctl daemon-reload
$ sudo systemctl enable elasticsearch
$ sudo systemctl start elasticsearch
$ sudo systemctl status elasticsearch
● elasticsearch.service - elasticsearch server
     Loaded: loaded (/etc/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-04-19 03:59:05 UTC; 52min ago
   Main PID: 493 (java)
      Tasks: 87 (limit: 9425)
     Memory: 4.5G
     CGroup: /system.slice/elasticsearch.service
             ├─ 493 /home/ubuntu/elasticsearch-8.7.0/jdk/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=/home/ubuntu/elasticsearc>
             ├─1077 /home/ubuntu/elasticsearch-8.7.0/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.securi>
             └─1241 /home/ubuntu/elasticsearch-8.7.0/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Kibana 8.7.0 설치

1. Kibana 8.7.0 설치 (https://www.elastic.co/kr/downloads/past-releases/kibana-8-7-0)

2. 설정파일 수정

$ wget https://artifacts.elastic.co/downloads/elasticsearch/kibana-8.7.0-linux-x86_64.tar.gz
$ tar -zxvf kibana-8.7.0-linux-x86_64.tar.gz
$ cd kibana-8.7.0

$ vi ./config/kibana.yml
# EC2에서 접속 가능하도록 kibana host주소 0.0.0.0 으로 설정

$ ./bin/kibana
[2023-04-11T09:21:08.539+09:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui]
[2023-04-11T09:21:15.693+09:00][INFO ][plugins-service] Plugin "cloudChat" is disabled.
[2023-04-11T09:21:15.695+09:00][INFO ][plugins-service] Plugin "cloudExperiments" is disabled.
[2023-04-11T09:21:15.696+09:00][INFO ][plugins-service] Plugin "cloudFullStory" is disabled.
[2023-04-11T09:21:15.697+09:00][INFO ][plugins-service] Plugin "cloudGainsight" is disabled.
[2023-04-11T09:21:15.702+09:00][INFO ][plugins-service] Plugin "profiling" is disabled.
[2023-04-11T09:21:15.752+09:00][INFO ][http.server.Preboot] http server running at http://localhost:5601
[2023-04-11T09:21:15.795+09:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
[2023-04-11T09:21:15.798+09:00][INFO ][preboot] "interactiveSetup" plugin is holding setup: Validating Elasticsearch connection configuration…
[2023-04-11T09:21:15.858+09:00][INFO ][root] Holding setup until preboot stage is completed.


i Kibana has not been configured.

Go to http://localhost:5601/?code=905398 to get started.

...

3. Kibana 실행

$ ./bin/kibana

Kibana 서비스 등록

$ adduser kibana
$ chown -R elastic:elastic /home/ubuntu/kibana-8.7.0
$ sudo vi /etc/systemd/system/kibana.service
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[Unit]
Description=kibana server
Wants=syslog.target network.target network-online.target
After=syslog.target network.target network-online.target

[Service]
RuntimeDirectory=kibana-8.7.0
WorkingDirectory=/home/ubuntu/kibana-8.7.0

LimitMEMLOCK=infinity
LimitNOFILE=65535
LimitNPROC=4096

ExecStart=/home/ubuntu/kibana-8.7.0/bin/kibana
ExecReload=/home/ubuntu/kibana-8.7.0/bin/kibana
RestartSec=3s

User=kibana
Group=kibana

[Install]
WantedBy=multi-user.target
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
$ sudo systemctl daemon-reload
$ sudo systemctl enable kibana
$ sudo systemctl start kibana
$ sudo systemctl status kibana
# Active: active (running)

접속 테스트

1. Elastic Search 주소 : https://127.0.0.1:9200/

2. Kibana 실행시 로그에서 표시된 Kibana go to주소 : http://localhost:5601/?code=905398

3. 접속 후 Elastic Search 시작시 나타난 enrollment token 입력 후 Apply

4. 목표

- 검색앤진 개발
- AI 개발

'Search Engine' 카테고리의 다른 글

Kafka Install  (0) 2023.07.04
Node  (0) 2023.04.19
Python  (0) 2023.04.11
NiFi  (0) 2023.04.10
  • 네이버 블로그 공유
  • 네이버 밴드 공유
  • 페이스북 공유
Search Engine 관련 글
글 더보기

티스토리툴바